Skip to content

Experimental performance family (default-off): load-balance infrastructure, active-box windowing, block-structured AMR, hybrid WENO/Riemann sensors#1628

Open
sbryngelson wants to merge 174 commits into
MFlowCode:masterfrom
sbryngelson:up/mega

Conversation

@sbryngelson

@sbryngelson sbryngelson commented Jul 3, 2026

Copy link
Copy Markdown
Member

Summary

An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is s_mpi_decompose_computational_domain, refactored through the new m_box module (byte-identical; covered by the existing suite).

  • Load-balance infrastructurem_box (partition arithmetic), m_load_weight/load_weight_wrt (per-cell load-weight field + imbalance metric), m_sfc_partition/sfc_partition_wrt (Morton-SFC predicted-imbalance diagnostic), m_load_balance/load_balance (weighted static decomposition at init; AMR-fine-work-aware), m_rank_timing/rank_time_wrt (per-rank compute-time diagnostic).
  • Active-box windowingm_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.
  • Hybrid reconstruction/flux sensorshybrid_weno and hybrid_riemann (+hybrid_weno_eps, hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).
  • Block-structured AMRm_amr + m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (blocks may span rank seams; fine halo exchange; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.

Full algorithm and user documentation: docs/documentation/amr.md (support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).

AMR physics support matrix (abridged; authoritative table in amr.md)

Supported and golden-tested: single- and multi-fluid (5-eq, mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-block pb/mv side-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) · active_box (blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).

Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) · amr_subcycle under IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).

Validation evidence

  • Conservation: reflux-exact (~1e-15 defects) across single/multi-fluid, viscous, bubbles, chemistry, phase change, through subcycled + regridding advances.
  • Free-stream preservation: exact (0.0) with subcycling + regrid armed, including 3D, IGR, and Lagrangian configs.
  • Decomposition exactness: np=1 == np=2 element-exact with blocks spanning rank seams (fields byte-identical incl. chemistry species).
  • Methodology: every feature validated by AMR-vs-no-AMR reference comparison with a control experiment calibrating the expected error scale, plus mechanism evidence from run logs (regrid counts, box coordinates, ghost-point counts) — a golden that does not mechanically exercise its path is treated as no coverage. Liveness goldens for the hybrid sensors are deliberately sized so a silently dead sensor fails them (~5e-4 signal vs 5e-5 tolerance) — they caught a real dead-sensor bug on Cray GPU during CI hardening.
  • GPU: full test suite with post-process on 2×V100 — OpenACC 644/644 executed pass, OpenMP offload 643/644 (one slurm launch flake). Speed spot-check (800² 2D blast, 2 ranks): AMR + dynamic regrid costs +3% wall over no-AMR at these sizes; ACC ≈ 12% faster than OMP; acc/omp solutions bitwise identical.
  • CI: one fully green 85-lane matrix on record (all GitHub-hosted lanes: gnu/intel × debug/no-debug/reldebug × double/single/mixed, macos, and the NVHPC 23.11–26.3 compat matrix). Fixes landed during hardening were each reproduced locally in the exact failing configuration first (intel FP drift, single-precision tolerance classification, gfortran bounds-check OOB, OpenACC present-table mapping ×3).

Known issues (all non-gating or in progress)

  • Frontier/Cray, in progress: two device-mapping issues in the IB machinery under CCE (plain-IBM descriptor corruption on gpu-omp; a present-table crash in the AMR-IB swap on gpu-acc). Being fixed with direct Cray iteration; the CCE bench/case-opt lane failures share the same root.
  • NVHPC 24.1/24.3 compat lanes (informational, continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK, -tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.
  • Accuracy postures documented rather than hidden: multi-fluid alpha-sum deviation bounded ~5.7e-3 at historical block faces during shock crossing (non-growing); viscous prolongation-ghost seam ~1e-6 np-dependence; IGR seam conservation is truncation-order (no reflux capture from the fused IGR kernels — exact-conservation capture is noted future work).

Review guide

The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).

All parameters ship default-off with case_validator entries, runtime checker gates, and case.md/amr.md documentation.

…es expand to fully contain each body plus margin (partial coverage is untested regime), overlapping expansions merge with size-cap and acoustic-conflict aborts, fine IB state rebuilds from geometry after every regrid; moving bodies stay gated; golden 7FC2F9F8 (verified: regrid fires, box contains body bbox, post-regrid rebuild runs; 3-golden IB battery clean)
…nd by exact parent-cell bisection (unified formula: boundary k belongs to parent isect_lo+floor(k/2), even=parent midpoint, odd=parent right edge) replacing the locally-uniform continuation; grid-uniformity aborts relaxed to arm the per-swap WENO coefficient recompute; golden F0DDE1B4 (stretched + dynamic regrid + restart, block parents span 2.08x spacing). Validated: 15-golden uniform battery bit-identical, stretched AMR-vs-reference errors match the uniform control (rho 7.3e-3 vs 7.1e-3 rel-L2; static, dynamic-regrid, and 2-rank), restriction integral error exactly 0; docs matrices updated incl. stale IB dynamic-regrid rows from f3814e5
…d-combination gates; the sensor arrays are coarse-idwbuff sized (fine extent guard keeps fine bounds inside) and the sensor recomputes from the live swapped idwbuff every RHS call, so each level evaluates its own sensor. Validated: AMR-vs-reference error scale matches the plain control to 3 digits on an acoustic case where the sensor is live (mixed verdicts, 56/72 central). Also found the existing hybrid goldens cannot detect a dead sensor (bitwise-identical to plain WENO by design at physical eps: only constant/eps-dominated cells go central and any convex weights reconstruct those identically - verified by forcing all-central, 2e-2 shift); added consequential-eps liveness goldens (eps 0.5 > shock phi ~0.3, dead sensor moves the answer ~5e-4): BA4340EA 60739A3E and AMR combos 053C5DDA DDC4BA8A
…e containment expansion reads the body's LIVE centroid (bbox helper factored from the expansion), the initial block gets the same body expansion when regridding is active, and a per-substage guard aborts if a moving body reaches its block boundary between regrids (consecutive contained positions keep every sub-time interpolate contained). Validated: 200-step tracking run (29 regrids, body rises 25 cells past its initial box, completion is the guard's own proof of following), guard-trip negative test aborts with the named message, error at/below the validated static-block control (rho L2 5.6e-3 vs 8.5e-3, identical body-surface Linf); Mach ~0.85 wakes legitimately exceed the per-rank box cap (named abort). Golden E72953D1 (Mach 0.25, 4 regrids)
…s - regrid suppresses tags and clips candidate boxes around the cloud's padded bbox (positions + mapCells smearing + stencil + drift margin, recomputed collectively per regrid), the fine advance skips the EL hooks (a bubble would map to wrong indices on the swapped grid), and a per-stage guard aborts if the cloud reaches an active block. FIX: EL volume fractions sum to the local liquid fraction (beta), not 1 - the multi-fluid sum-to-one prolongation closure corrupted the EL state (found by the free-stream battery: O(1) momentum injection at block edges in quiescent flow; exactly 0 after the exemption); EL is correspondingly exempt from the mpp_lim gate. Validated: free-stream through block exactly 0, guard-trip named abort, 800-step wave + two-way bubble at 1e-9 field consistency with the bubble trajectory identical; goldens 4B08E9B7 (static) BCE1BBAE (dynamic regrid); multi-fluid AMR battery still green (closure fires for non-EL)
…ing - the fine block runs its own fixed-iteration sigma solve seeded and Dirichlet-bounded by the converged coarse sigma (piecewise-constant parent injection over the full buffered range; per-iteration BC/halo populate skipped under amr_in_fine_advance; coarse jac/jac_old warm-start state bounced across the fine advance), fine RK update uses the IGR coefficient form (dt embedded in the rhs), flux-register capture/apply return early under igr (no face fluxes exposed by the fused IGR kernels - seam conservation is truncation-order, documented), amr_subcycle gated. Validated: free-stream through block EXACTLY preserved, AMR-vs-reference at resolution scale (rho 1.3e-4 rel-L2, static and dynamic regrid), transverse seam artifact 3e-4 absolute from the coarse/fine sigma jump (truncation-level, documented), conservation drift at the reference's own outflow scale. Goldens 6C20B752 (static) 660FFBFE (dynamic regrid)
…mr_cur)%pb_f/mv_f as assumed-shape actuals made nvfortran emit a component-section data clause that dies partially-present on device (CUDA illegal address; caught by the full-suite GPU runs on both acc and omp - CPU passes only because the fine/coarse lbounds coincide); the ghost fill now selects its slot target by integer and references the slot arrays directly inside the kernels (the proven pattern of backup/update/lerp/restrict), and the pb/mv prolongation is now a device kernel reading the device-current coarse side-state (the old host-side copy would read a stale host mirror at regrid time on GPU). CPU nonpoly battery bit-identical (4/4 goldens)
…P_SFs pattern - the previous fix removed the ghost-fill component-section clause but the fine s_compute_rhs call still passed raw amr_slots(amr_cur)%pb_f/mv_f/rhs_* 5D members as actuals (second CUDA illegal address on device); persistent slot state (pb/mv f, stor, ghost_a/b) is now type(pres_field) with %sf allocated + ACC_SETUP_SFs per slot (the exact pb_ts pattern the coarse solver passes through the same RHS interface on GPU), and the fine rhs scratch moves to shared module arrays amr_rhs_pb_f/amr_rhs_mv_f (slots advance sequentially; mirrors the coarse rhs_pb pattern). CPU nonpoly battery bit-identical (4/4)
…on stays host-side - the trace (NV_ACC_NOTIFY) pinned the remaining illegal address to the device prolong kernel referencing pb_ts(1)%sf across modules; ALL pbmv kernels (fill/backup/rk-update/lerp/restrict) now receive the slot and coarse arrays as assumed-shape dummies with %sf pointer-member actuals - the one device-proven pattern (the coarse pb chain and the fine q_cons machinery both pass everything as arguments) - and the prolongation reverts to the original host loops + device push (both call paths make the coarse host mirrors current first: init writes them on the host, regrid refreshes them from the device); also fixes mv_f missing %sf in the two regrid GPU_UPDATE lists after the pres_field conversion. CPU nonpoly battery bit-identical (4/4)
…t-in (honor_io_keys), not content-based - honoring any case's explicit parallel_io/prim_vars_wrt broke the 18 Example-derived goldens whose imported case.py sets those keys (goldens were generated under the clobber): every full CI test lane failed with line-count mismatches (first seen on intel-reldebug only because it finishes first; reproduced locally under nvfortran, exit 0 with 6 extra prim.*.dat pack entries). The two tests the exemption exists for (5EFB3277 MPI-IO AMR restart, 0253D658 load_balance) opt in via define_case_d(honor_io_keys=True); UUIDs and goldens unchanged. Validated: ibm_ellipse/ibm_stl/phasechange_bubble pass again and both opt-ins keep their coverage (5/5 with -a)
…-clip verification after body expansion/merge ran only under acoustic_source, so an expanded box could silently regrow over the cloud in ib+lagrangian configs (fail-closed check now covers both exclusions); (2) the per-stage cloud guard ran only on block-owner ranks, missing bubbles whose smearing support reaches a block across a rank seam (now every rank checks its local bubbles against the global block box); (3) the moving-body containment guard was skipped entirely on static blocks (now always on for moving bodies, margin 0 there since validated static configs sit tighter than the regrid margin); (4) the position-to-cell-index conversions (int((x-beg)/dx(0))) in the IB expansion/guard and cloud exclusion assume uniform spacing and dx(0) is rank-local on stretched grids (rank-inconsistent regrid boxes) - stretched+lagrangian and stretched+ib-dynamic-regrid now fail closed at init (checker+validator). Comment/docs truth pass: fill_fine_ghosts_pbmv doc described the reverted selector design, nonpoly QBMM rows still said regrid/subcycle gated (both supported), amr.md mandatory settings said WENO-only (IGR allowed), recompute-flag comments still axisym-only. New goldens close the stretched coverage gaps: 79B334C7 (2-rank stretched: first exercise of the start_idx offset terms where a wrong offset changes coordinates), B7704247 (2D stretch_y: first exercise of the y-direction bisection formula; 2 regrids, block spans 65% y-spacing variation). 20-golden battery green incl. all touched features
…1e-12 relative threshold sits below single-precision grid roundoff (~1e-7), classifying EVERY grid as stretched under --single; harmless when it only armed the WENO recompute (wasted work), but the new stretched-combo gate turned it into a spurious init abort for the Lagrangian+AMR goldens on the CI single lane (reproduced locally). 1e3*epsilon(1) ~ 1e-13 double / 1e-4 single, both above roundoff and far below real stretching ratios (1.1-2x). Single: EL pair + stretched golden pass; double: 8-golden battery green
…R - the IGR rhs kernels write over -1:m+1 in EVERY dim including collapsed ones (the coarse rhs_vf is allocated (-1:m+1,-1:n+1,-1:p+1) under igr), but the fine slot rhs used the buffered fine extents whose collapsed dims are 0:0 - an out-of-bounds write in every 2D IGR AMR fine stage, trapped by gfortran bounds checking on the CI reldebug lane (silent in no-debug builds: the stray writes landed outside the read range, goldens unaffected). Buffered extents already cover -1:+1 in active dims. Validated: both IGR AMR goldens pass under reldebug bounds checking and the no-debug battery is unchanged
@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 54.09742% with 801 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.47%. Comparing base (f4ab1aa) to head (ba8f964).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
src/simulation/m_amr_registers.fpp 61.98% 66 Missing and 64 partials ⚠️
src/simulation/m_riemann_solver_hllc.fpp 43.01% 85 Missing and 17 partials ⚠️
src/simulation/m_sfc_partition.fpp 1.98% 97 Missing and 2 partials ⚠️
src/simulation/m_weno.fpp 66.20% 54 Missing and 19 partials ⚠️
src/simulation/m_load_balance.fpp 44.00% 48 Missing and 8 partials ⚠️
src/post_process/m_data_input.f90 62.40% 38 Missing and 12 partials ⚠️
src/simulation/m_load_weight.fpp 3.84% 48 Missing and 2 partials ⚠️
src/simulation/m_mpi_proxy.fpp 40.47% 45 Missing and 5 partials ⚠️
src/simulation/m_data_output.fpp 0.00% 25 Missing and 5 partials ⚠️
src/common/m_mpi_common.fpp 44.23% 16 Missing and 13 partials ⚠️
... and 13 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1628      +/-   ##
==========================================
+ Coverage   60.43%   60.47%   +0.03%     
==========================================
  Files          83       91       +8     
  Lines       19871    22724    +2853     
  Branches     2956     3646     +690     
==========================================
+ Hits        12010    13743    +1733     
- Misses       5860     6632     +772     
- Partials     2001     2349     +348     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…e kernels referenced sw_idwbuff (and amr_isect_lo/start_idx) directly in loop bounds and bodies; those are host-only module state, so OpenACC's present lookup failed fatally (data in PRESENT clause not found: sw_idwbuff) on BOTH Cray-acc (Frontier) and nvfortran-acc (Phoenix) while OpenMP's implicit map(to) tolerated it - exactly why only acc lanes crashed on the IGR AMR goldens. Root-caused with NV_ACC_NOTIFY on a V100 (fault after the s_amr_copy_fine_fields launch, present-table dump names sw_idwbuff). Scalar hoisting is the established kernel idiom everywhere else in the file. CPU pair bit-identical
…ic golden get override_tol=1e-8 - the consequential-eps regime deliberately amplifies reconstruction differences, which also amplifies benign backend FP variation: intel -O3 drifts these five goldens 2e-12..3.3e-10 rel past the 1e-12 default tolerance (intel no-debug CI lane), while the signal the hybrid goldens exist to catch (a dead sensor) moves the answer ~5e-4, four orders above the new tolerance. override_tol bypasses the --single 1e8 relaxation, so the four hybrid goldens join the --single skip list (the acoustic one already matched an existing skip pattern). UUIDs and goldens unchanged; gnu double 5/5
…-gating CI quirk at the EL AMR golden definitions - the exact failing stack (NVHPC 24.3 SDK, -tp=px -Kieee, HPC-X MPI, and the CI docker image itself run via apptainer) passes on Phoenix, as do native and zen2-targeted builds; suspected runner-hardware interaction with old nvfortran codegen, lanes are continue-on-error, 24.5+ green. Comment-only; UUIDs unchanged
… - the first intel no-debug failure only reported the FIRST out-of-tolerance variable (3.3e-10 rel), undersizing the retune; the next matrix run showed the true intel -O3 drift reaches 3.1e-7 rel on the AMR+hybrid pair. 1e-6 keeps >2 orders of margin above the drift and >2 orders below the ~5e-4 dead-sensor signal these goldens exist to detect. Acoustic golden stays at 1e-8 (its 3.3e-10 drift held). Comparison-side only; goldens/UUIDs unchanged; gnu 5/5
…sor engages - hybrid_weno/hybrid_riemann/hybrid_weno_eps/hybrid_smooth_flux are read inside the WENO reconstruction and HLLC flux device kernels but were in no GPU_DECLARE, so on Cray they read undefined on-device and the sensor ran vacuously (all full-WENO); nvfortran tolerated the unmapped module scalars, CCE did not. Added them to the not-case-opt GPU_DECLARE(create) and the matching GPU_UPDATE(device) beside the wenojs peers. Same class as f5eea61, opposite direction (flags not bounds). Verified: the 4 consequential-eps hybrid liveness goldens (60739A3E/BA4340EA/053C5DDA/DDC4BA8A) fail->pass on Frontier Cray-acc
… ICFL monitor sees valid data - active_box narrows the cons->prim producer to the light-cone box, leaving the frozen ambient exterior primitives (converted velocity/pressure) uninitialized; the full-domain ICFL/vCFL stability monitor then reads that exterior and NaN'd (fatal MPI_Abort on Cray-cpu Release, silently finite on gfortran). The first active_box RHS now runs the full-domain conversion once (ab_prim_seeded) to seed the frozen exterior, then narrows as before - the exterior is never overwritten, so all full-domain consumers (ICFL, probes, IB) stay valid. Optimization preserved (one extra full conversion at startup). Verified: ECABA006 fail->pass on Frontier Cray-cpu
…s; drop the redundant copyins - ghost_points is GPU_DECLARE'd, so @:ALLOCATE establishes its mapping and the explicit GPU_ENTER_DATA(copyin) layered a second dynamic map on the declared entry (CCE-OMP corrupts its descriptor: lib-4425 at first coarse setup, ALL IBM tests on Frontier gpu-omp), and the fine-setup path's BARE deallocate/allocate leaked the present-table entry so a later allocation reusing the host address range failed partially-present on Cray-acc (AMR-IBM only: the collision needs a SECOND setup, i.e. regrid). Contents were always filled by device kernels, so the copyins uploaded nothing. The codebase already documents the bare-deallocate leak at the finalize site. CPU IB battery 5/5; targets the two open Frontier items (B, A-IB) - Cray verification pending on Frontier
…y decision theory - the harness reports only the FIRST variable past tolerance, so the three prior retunes (1e-12 default -> 1e-8 -> 1e-6) each just revealed the next quantile of the same deterministic deviation field: intel-CPU and Cray-acc produce IDENTICAL sensor-on results differing from the gnu-generated goldens by up to ~6e-6 (compiler FP-contraction behavior on legitimate results, not noise; nvfortran stays under 1e-6). 5e-5 sits an order below the ~5e-4 dead-sensor signal these liveness goldens exist to catch and 8x above the worst observed drift; if any platform ever exceeds 5e-5 the static-golden design itself stops discriminating and should be redesigned rather than retuned. gnu 5/5
…ctive window (init containment abort + regrid window-clamp: the windowed coarse RK update would silently drop reflux corrections at faces outside the window), and the fine advance swaps ab_active off (window bounds are coarse cell indices, meaningless on the swapped block grid; a contained block is all-active by definition). The combination is np=1 by active_box's own MPI gate, the window's monotone growth means containment established at init/regrid holds between them, and the frozen exterior is valid ambient data for ghost prolongation (its primitives seeded since f55ccb9). Validated: ab+AMR vs plain AMR agree to 9.8e-15 over 200 steps including the window self-disable transition (the active_box round-off spec, with the nonzero diff proving the windowing engaged), dynamic-regrid pair bitwise identical with 4 regrids each, containment abort negative-tested with the named message. Goldens 4364FA6B (static, window live all 10 steps) 8AEA60DD (dynamic regrid)
…nd quantified the failure: B and the GLM psi field ride the generic conservative machinery structurally (the fine advance, prolongation, reflux, and restriction all compile and run), but per-component prolongation/reflux is not divergence-preserving, and on a magnetized 2D Brio-Wu (Bx 0.75, By flip, HLL + hyper_cleaning) the coarse/fine seam acts as a CONTINUOUS O(1) monopole source that cleaning spreads but cannot remove: max|divB| 0.53 in the block interior and 0.36 far-field vs the no-AMR run's 1.4e-3 cleaning background, despite field L2 errors looking normal (1e-2 class). HLLD NaNs outright at step 158 - and separately, m_riemann_solver_hlld has NO GLM coupling at all, so hyper_cleaning+HLLD is silently inert everywhere (upstream latent gap, flagged to maintainers in the PR). Supporting MHD needs constrained-transport-class B prolongation/reflux (future work). Gates restored with the measured rationale in checker/validator/docs; upstream MHD battery green (B=0 and magnetized goldens unaffected)
@sbryngelson

Copy link
Copy Markdown
Member Author

Upstream latent gap found during the MHD+AMR investigation (independent of this PR): m_riemann_solver_hlld.fpp contains no GLM terms, so hyper_cleaning = T with riemann_solver = 4 (HLLD) is silently inert — psi never evolves and no cleaning occurs. The existing hyper_cleaning golden uses HLL, which does carry the GLM coupling, so CI never sees this. Worth either adding GLM terms to HLLD or a validator prohibit on the combination.

Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1).

…y By/Bz (Bx is the uniform Bx0 parameter), so div(B) is IDENTICALLY zero and the measured 2D/3D seam-monopole failure mode is structurally absent: By/Bz reflux and restrict as ordinary conserved scalars, and HLLD (the div-brittle solver that NaN'd in 2D) is stable. Gates narrowed from all-MHD to n > 0 (checker + validator, with the 1D exemption rationale). Validated: 1D Brio-Wu HLLD (static + 40-regrid dynamic) and 1D relativistic RMHD, AMR-vs-reference at resolution scale (rho 0.8-1.6e-2 rel-L2 on a 200-cell tube, matching hydro controls), Bz preserved exactly. Goldens 73355E90 (HLLD static) 4B25CC24 (HLLD dynamic regrid) CBDF2538 (RMHD); docs matrix split into 1D-supported / 2D-3D-gated-with-measurements rows
…with the measured root cause - attempted lifting this gate and found the position-to-index conversion (the original stated reason) is only the SURFACE blocker and is fixable (global-boundary bisection, allreduce-MAX assembled, rank-consistent - uniform goldens bit-identical under it). The real blocker is deeper: IB/Lagrangian floor buff_size to 10/6, but s_amr_recompute_weno_coefs (armed only on nonuniform grids, so never exercised while this combo was gated) indexes poly_coef_cb* over -buff_size:m+buff_size while m_weno sized those arrays with a smaller buff_size at init -> out-of-bounds write in s_compute_weno_coefficients. Needs the WENO coef arrays sized to the final buff_size before the gate can lift. Comment-only; gate unchanged (correctly still fail-closed)
…d) - decouples the fine level's ownership from the coarse mirror decomposition, the AMReX per-level DistributionMapping idea. amr_block_owner(:) assigns each active block a single owner rank by chains-on-chains balancing of fine-work weight in Morton (Z-order) block order - deterministic from the replicated block geometry, no communication. Computed at init and each regrid, reported as a predicted fine-work imbalance, but NOT yet applied: the mirror amr_owns_all still governs ownership, so behavior is unchanged (23/23 AMR goldens bit-identical; diagnostic shows 1.00x single-rank, 2.00x on a granular 2-block/2-rank case). Phase 2 switches amr_rank_owns_block to the owner map + adds the coarse<->fine gather/scatter (the correctness-risk checkpoint). Design: docs/documentation/amr_fine_distribution.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants